Systems and methods for optimizing a scene

ABSTRACT

Systems and methods for optimizing properties of objects within a scene or achieve a visual goal.

CROSS REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional PatentApplication No. 61/102,625, filed Oct. 3, 2008, the disclosure of whichis incorporated by reference herein in its entirety.

RELATED APPLICATIONS

This application is also related to U.S. Provisional Patent ApplicationNo. 61/102,618, filed Oct. 3, 2008 and U.S. Provisional PatentApplication No. 61/102,670, filed Oct. 3, 2008.

BACKGROUND

The human visual system is a capacity limited system in that it can onlyprocess a relatively small number of objects at any given time. This istrue, despite the fact that there are many objects that may be visibleat any given time. From the array of objects visible to a human, thathuman's visual system will only attend to, or processes, one (or veryfew) objects at any given time. When a human looks at an image or ascene, his visual system will shift attention (and mental processing)from one object to another.

There has been a substantial amount of research in the area of humanvisual attention. This research has generated numerous studies directedtoward understanding the behavior of human visual attention, as well asmany computational models of visual attention. These computation models(sometimes called visual attention models, eye-gaze prediction models,attention models, or saliency models) predict where, given visualstimuli (for example, a picture or a scene), a person will allocatetheir visual attention or gaze.

These models provide predictions about the objects or regions within thescene that will attract visual attention. Typical real world scenes,however, are often highly dynamic.

The image projected to the human will change when, for example, theperson's vantage point changes, the objects within a scene changepositions or orientation, or the lighting changes (casting differentshadows). Furthermore, the observer himself may introduce uncertaintyinto the predictions (the observer may be pre-occupied, or otherwisedisposed to a particular attention pattern). Any variability in theimage projected from a scene, or variability across observers, or evensmall changes to the scene itself, can significantly change thepredictions made by these models. This can be problematic when usingvisual attention models in applied settings.

SUMMARY

Systems and methods for optimizing properties of objects within a sceneto achieve a visual goal. In exemplary embodiments, costs are associatedwith permissible changes to the scene, then a multitude of scenes aregenerated based on the universe of permissible changes. These multitudeof scenes are then analyzed using a visual attention model to determinescenes that meet the visual goal, and possible the costs associated withchanges that give rise to the multitude of scenes. In other exemplaryembodiments, rewards may be associated with particular visual goals, anda search made of the multitude of scenes that maximize the reward whileminimizing the cost. In a multi-perspective context, rewards could bedefined in terms of the vantage point from which a potential observershould see an object. In some exemplary embodiments, robustness may beused in the scene optimization routines. For example, the visual goalmay be defined in terms of robustness (a particular object is desired tobe robust), or scenes that are determined to meet a particular visualgoal may subsequently be evaluated for robustness (with a bias, forexample, toward the robust scenes that meet the visual goal). In yetmore exemplary embodiments, the amount of time an object is viewable maybe considered by the scene optimization system.

In one embodiment, a computer-implemented method is described,comprising: defining at least one visual goal for a scene, the scenecomprised of a plurality of objects, the visual goal specifying at leastone object in the scene which is desired to be predicted to be attendedto by a visual attention model; receiving input defining allowablechanges to the scene; assigning a cost estimate to at least some of theallowable changes; automatically modifying some of the plurality ofobjects consistent with the defined allowable changes so as to producemodified scenes; evaluating the modified scenes with a visual attentionmodel; and, determining, based on the evaluation, at least some of themodified scenes that achieve the at least one visual goal for the scene.

In another embodiment, a computer-implemented method is described,comprising: defining at least one visual goal for a scene, the scenecomprised of a plurality of objects, the visual goal specifying at leastone object in the scene which is desired to be not predicted to beattended to by a visual attention model; receiving input definingallowable changes to the scene, which define how at least some of theplurality of objects may be changed in order to achieve the goal;assigning a cost estimate to at least some of the allowable changes;automatically modifying some of the plurality of objects consistent withthe defined allowable changes so as to produce modified scenes;evaluating the modified scenes with a visual attention model; and,determining, based on the evaluation, which of the modified scenesachieves the at least one visual goal for the scene.

In another embodiment, a computer system comprising: a processor andmemory; a scene optimization module operative to at least: (1) receiveinput defining at least one visual goal for a scene, the scene comprisedof a plurality of objects, the objects having properties; (2) receiveinput defining allowable changes, which define how properties can bechanged; (3) associate allowable changes with a cost estimate; (4)generate modified scenes by modifying the properties of objectsconsistent with the allowable changes; (5) interact with a visualattention module to determine which modified scenes meet the visualgoal; and, (6) determine the costs associated at least some of themodified scenes; a visual attention module operative to evaluate a sceneand predict objects within the scene that will tend to attract visualattention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a flowchart illustrating high-level functioning of arepresentative visual attention module.

FIG. 2 is a sketch of a representative scene.

FIG. 3A is an artist's rendering of output from a visual attentionmodule.

FIG. 3B is an artist's rendering of output from a visual attentionmodule.

FIG. 4 is a graph showing properties of two example objects.

FIG. 5 shows how a billboard object can be set against differentbackground scenes.

FIG. 6 is a diagram of functional modules in a system for evaluating therobustness of a scene, or objects within a scene.

FIG. 7 is an illustration showing example scene modifications.

FIG. 8 shows an alternative embodiment of the visual attentionrobustness assessment system, in which the visual attention robustnessassessment system additionally includes web server module.

FIG. 9 is a high-level flowchart illustrating one embodiment of theprocess the visual attention robustness assessment system may employ toevaluate the robustness of an object within a scene or a scene itself.

FIG. 10 is a flowchart showing the high level process of sceneoptimization.

FIG. 11 is a graph illustrating two different reward structures as afunction of the serial position of the attentional saccade to the objectof interest.

FIG. 12 is a diagram of functional modules in a system for sceneoptimization.

FIGS. 13A and 13B are an artist's rendering of two respective scenes.

FIGS. 14A and 14B are an artist's rendering of two respective scenes.

FIG. 15 is a high level flowchart showing multi-perspective sceneanalysis.

FIG. 16 is a diagram of a conference room.

FIGS. 17A and 17B show different perspectives of the conference room.

FIG. 18 is a graph illustrating three different example analyses done tovarious objects in the conference room.

FIG. 19 is a diagram of a multi-perspective scene analysis system.

DETAILED DESCRIPTION

Visual Attention Models

Visual attention models determine the extent to which regions within ascene differ with respect to the likelihood that they will attractvisual attention or attract eye movements. A scene is any form of input(for example, graphical image) that is amenable to evaluation by avisual attention model, and may, for instance, be a digital photograph,a virtual 3D scene, a web page, a document, or a video.

A visual attention model's disposition to an object refers to how amodel of visual attention characterizes the relative saliency of anobject within a scene. For example, some visual attention models willsuperimpose a trace line around predicted objects. Other visualattention models will generate heat maps that may be superimposed overthe image or viewed separately from the image. Even others may generateand assign a value to a particular object and/or region, with the valuerepresenting that object's saliency in relative terms. In the context oftrace lines, the disposition of an object may be considered as“selected” (when traced) or “not selected” by the model. In the contextof heat maps, the disposition of an object is the degree to which thealgorithm has selected the object (or not selected the object). And inthe context of a saliency number, the disposition of the object may bethe saliency number itself.

There are many visual attention models that predict where within a scenehuman visual attention will be allocated. Generally, these visualattention models take a single image of an environment as input andgenerate predictions about where attention will be allocated within thatscene. Empirical approaches expose human subjects to a scene and tracktheir eye movement. This empirical approach is resource intensive,however, so a number of mathematic models have been developed thatattempt to predict attention by analysis, at least in part, of thescene. That said, empirical approaches could also be used as part of thesystems and methods described herein, and as used herein an empiricalapproach is deemed to be a type of visual attention modeling.

The basic methodology of one of these models is represented in FIG. 1,which is that proposed by Itti, L. & Koch, C. (2000), A saliency-basedsearch mechanism for overt and covert shifts of visual attention, VisionResearch, vol. 40, pages 1489-1506. At a high level, FIG. 1 shows how aprediction is made for visual attention by assessment of “bottom-up”features such as color, motion, luminance, edges, etc. which serve asbuilding blocks of the visual representations mediating some aspects ofhuman vision. First, a scene, in the form of a digital photograph, isprovided to a computer-implemented version of the Itti and Koch model(step 10). Next, a feature extraction process analyzes the digitalphotograph for colors, intensity, orientations, or other scene cues,such as motion, junctions, terminators, stereo disparity, and shape fromshading (step 11). The feature extraction process yields a plurality offeature maps (step 12), which are combined to produce a saliency map(step 13). In the case of the Itti and Koch model, the saliency data isprovided to a user as a rendering of the original digital photographwith the “brightest” objects being to which the model has predictedvisual attention will be next allocated. This predicted object isidentified to be visually salient (step 14) in a “winner-take-all” typealgorithm (step 15), and the process then repeats until a plurality ofobjects are identified by the model.

FIG. 2 is an artist's rendering of a scene 201 that could be provided toa visual attention model such as Itti and Koch. It is a simplified sceneincluded here for illustrative purpose only; in practice the scenes areoften actual digital photographs, or videos, and are much more complex.FIG. 2 includes a number of objects within the scene, such as the star202, flower 203, face 204, star 205, arrow 206, and cup 207.

FIG. 3A is an artist's rendering representing how output from the Ittiand Koch model could be represented. The highlighted (and in thisrepresentative illustration, encircled) objects are those that the modelpredicts to be visually salient. For example, star 202 is in this figurewithin highlight border 208; flower 203 is within border 209; face 204is within border 210; star 205 is within border 211; arrow 206 is withinborder 212; and cup 207 is within border 213. Thus the model in thisinstance has determined six objects that are, relative to other objects,more visually salient. This particular model also predicts how attentionwill move among the objects determined to be above some visual saliencythreshold. For example, visual attention pathway 301, 302, 303, 304, and305 show a predicted visual attention pathway.

FIG. 3B is a second artist's rendering showing a further manner in whichoutput from the Itti and Koch model is sometimes represented. Inaddition to what is shown in FIG. 3A, FIG. 3B includes the sequence ofpredicted visual attention. For example, star 202 is labeled “1”(attention sequence number 214), and flower 203 is labeled “2”(attention sequence number 215) and so forth.

Of course, FIG. 3A and FIG. 3B are just one manner in which a visualattention prediction may be conveyed to a user; different modelsrepresent this information (or some subset of it) in different ways. Forexample, not every model determines a predicted attention sequence,though such an attention sequence could be arrived at by determining theobject with the highest level of visual saliency, then eliminating thatobject and finding the next highest, etc.

Itti and Koch's model is representative of a “bottom-up” visualattention model, in that it makes its predictions based on analysis ofthe particulars of the scene. Other bottom-up visual salience models aredescribed in these references: Gao, Mahadevan, and Vesconcelos (2008).

Besides bottom-up models, there is another class of models referred toas “top down” models of visual attention. In contrast to bottom-upmodels, these models start with a scene and either an explicit task (forexample, avoiding obstacles and collecting objects) or prior knowledgeof the world that will influence where attention will be allocatedduring a specific search task (for example, chairs tend to be on thefloor and not on the ceiling). This knowledge (both task- andscene-based) is used in conjunction with the bottom-up features todirect attention to objects within the observed scene. Some exemplarytop-down models are described in Rothkopf, C. A., Ballard, D. H. &Hayhoe, M. M. (2007), Task and context Determine Where You Look, Journalof Vision 7(14):16, 1-20; and also in Torralba, A., ContextualModulation of Target Saliency, Adv. in Neural Information ProcessingSystems 14 (NIPS), (2001) MIT Press, 2001. For example, Torralba's modelof visual attention has prior knowledge about the features that comprisea particular type of object and information about the absolute andrelative locations of these objects within the scene. This priorknowledge provides “top-down” influences on searching for specifictargets within a scene.

The art has evolved to include hybrid visual attention models that havefeatures of both bottom-up and top-down design, and adapted fordifferences in the types of scenes the models will be exposed to (forexample video versus still images, outdoor images versus web pages, andso forth).

Robustness

Robustness refers to the sensitivity of predictions output from a visualattention model to either of, or some combination of:

-   -   (a) changes and/or variability within the visual properties of a        scene, which includes, for example, the arrangement of objects        within the scene, lighting of the objects, the color of objects,        etc. (referred to as “external variability”); or,    -   (b) changes and/or variability within an observer, or the model        of observation (referred to as “internal variability”).

The term object as use herein refers to either a thing or area or regionwithin a scene, as the case may be, or the regions within a scene thatare analyzed by a visual attention model. The term object is usedinterchangeably with “area” or “region” as the case may be.

The two types of changes (a) and (b) are referred to collectively asinternal or external, or 1E, changes, and various methods ofintroducing, then evaluating, such changes are discussed further below.

Robustness evaluation is a method for measuring the effect of IE changeson where attention will be allocated within a scene. Robustness of anobject within a scene, then, is the degree to which a visual attentionmodel's predicted disposition of a particular object changes, or doesnot change, despite IE changes.

The robustness of a group of objects is the degree to which a visualattention model's disposition to more than one object (a collection ofobjects) changes or does not change with IE changes.

Robustness of a scene is a measure of the degree to which the visualattention model's disposition of objects in the scene will change, ornot change, despite IE changes. For example, if a visual attention modelpredicts that attention or fixations will be made to the same fourobjects within a scene, and these four objects tend to remain predictedby the visual attention model despite IE changes, the scene tends to bemore robust than a if the four objects changed following IE changes.

Visual Attention Robustness Assessment System

FIG. 6 is a diagram showing exemplary functional modules in visualattention robustness assessment system 403, which is a system forevaluating the robustness of a scene, or the robustness of objectswithin a scene, or the robustness of objects across multiple scenes. Ofcourse, these methods may also be practiced manually. Visual attentionrobustness assessment system is, in the embodiment shown in FIG. 6,within computer system 408. Computer system 408 may be any generalpurpose or application-specific computer or device. It may be astand-alone laptop computer, or a plurality of networked computers.Further, computer system 408 may be a handheld computer, digital camera,or a tablet PC, or even a cellular telephone. Computer system 408, inone embodiment, has various functional modules (not shown in FIG. 6)that comprise an operating system. Such an operating system facilitatesthe visual attention robustness assessment system's access to thecomputer system's resources. Computer system 408 has a processor andmemory, and various traditional input/output interfaces.

Visual attention module 403 is any embodiment of any visual attentionmodel or combination of models. As mentioned earlier, there aredifferent types of visual attention models, but they all, to someextent, predict an object or area within a scene to which visualattention will tend to be allocated. Visual attention module 403 isshown in FIG. 6 as part of visual attention robustness assessment system402, but visual attention module 403 in another embodiment operates as astand-alone computer process or even as a service provided over any typeof computer network (such as the World Wide Web) at a remote computer.

VAM modification module 404 is the visual attention module modificationmodule. VAM modification module 404 modifies aspects of the visualattention module's parameters or architecture. This modification can byaccomplished in many ways depending on the implementation of the visualattention module 403. For example, visual attention module 403 mayitself support function calls that modify aspects of how the visualattention module works. In one embodiment, visual attention module 403may support a function call that modifies weighting given to certainaspects of the scene (brightness, for example). In another embodiment,if the visual attention module is invoked via command line, variousswitches could be employed to change variables that are within thevisual attention module. Alternatively, if the visual attention module403 is embodied in a script or programming code, the IE modificationmodule could modify the script or programming code itself. In anotherembodiment, the entire visual attention model is replaced by anothervisual attention model. The particular ways in which the VAMmodification module 404 modifies the visual attention module 403'sunderlying visual attention model (or the application of such a model toa scene) is discussed further below, but one example might be to modifythe weights associated with the plurality of feature maps that are usedto generate the saliency map. This could be accomplished through afunction call that would modify these values based upon a distributionof acceptable values.

Scene modification module 405 modifies the scene that will be providedto the visual attention module 403 for analysis. The scene changes arethe changes that are associated with the external environment thatsimulate the dynamic nature of the environment. Such externalvariability may include, for example, moving objects within the scene(for example, the placement of pedestrians), changes in shadows due tochanges in lighting direction, or changes in atmospheric conditions (forexample, dust in the air). This variability can be generated in a numberof ways. One method is to place a camera within the scene and capture animage at different times. This would capture the natural variability ofa real scene. Another method is to capture a single image of the sceneand specify possible changes that could be made to the individualelements and objects within that scene. An illustration of such atechnique is shown in FIG. 7. Scene 801 is a representation of anoriginal scene. Scene 802 shows the objects from scene 801 replaced withscene variability elements, such as shadows and other objects (forexample, cars or birds—any object that can be placed within the scene).Scene 803, 804, 805, and 806 show different combinations of the originalscene 801 with scene variability elements. One skilled in the art willappreciate there are uncountable different approaches to modifying ascene, such as using commercially available image editing software suchas that marketed by Adobe Systems of San Jose, Calif., under the tradename “Photoshop.” A few have been presented herein as non-limitingexamples only.

Scene modification module 405 adds external variability to a scene,which will in turn result in multiple versions of the scene, or sceneinstances. The collection of scene instances, in one embodiment,captures the variability within the scene. To measure the robustness ofan object or the robustness of the scene, each scene instance issubmitted to visual attention module 403 to produce a prediction as towhere visual attention will be allocated within each scene instance(that is, to which objects). Information pertaining to each run is savedin database 407, then robustness assessment module 409 evaluates thedata to generate statistics based upon objects to which the modelpredicted attention would be allocated, across these different instances(and as was described in further detail earlier).

Graphical user interface module 406 facilitates interaction with user401. Graphical user interface module 406 may, for example, call onoperating system resources (from computer system 408) to build agraphical user interface to solicit input from user 401. This input inone embodiment includes the location of the scene, as well as otheroperating parameters for the visual attention robustness assessmentsystem. This input, in one embodiment, would specify regions and/orlocations within the scene that user 401 is interested in evaluating. Inaddition to specifying such locations, user 401 could specify what typesof variability is to be considered by the visual attention module 403.This may include general or specific internal variability, or general orspecific external variability. For example, one specific type ofexternal variability might be changing the image due to variations inlighting direction. As light changes, the shadows produced will change.This is a specific type of external variability because it does notconsider other factors such as dynamic objects, atmospheric aberrations,and so forth. An example of general internal variability would be acondition in which the weights for each feature map are allowed to varyindependently. An example of specific internal variability is when theweights of one set of feature maps vary (for example, brightness) butthe others do not. Graphical user interface module 406 also facilitatesgetting input, possibly from the user, regarding from where images ofthe scene should be acquired. Possible locations include, for example, adatabase, or a flat file.

Robustness assessment module 409 controls the interaction of the othermodules in order to evaluate the robustness of objects within a scene orthe robustness of the scene itself. For example, the robustnessassessment module 409 invokes the visual attention module 403 asnecessary, as well as the VAM modification module 404 and the scenemodification module 405. The manner in which the robustness assessmentmodule 409 invokes the various modules may be modified by input fromuser 401, provided to the robustness assessment module 409 via, forexample, graphical user interface module 406. Robustness assessmentmodule 409 also evaluates data provided by other modules and generatesreports as necessary.

Database 407 handles the data storage needs of the visual attentionrobustness assessment system 402. Among other things, database 407 mayhold images of the scene. Database 407 may be any computer memory. Itmay be random access memory, or a flat file, or one or more databasemanagement systems (DBMS) executing on one or more database servers. Thedatabase management systems may be a relational (RDBMS), hierarchical(HDBMS), multidimensional (MDBMS), object oriented (ODBMS or OODBMS) orobject relational (ORDBMS) database management system. Database 407, forexample, could be a single relational database such as SQL Server fromMicrosoft Corporation.

User 401 is any user of the visual attention robustness assessmentsystem. In some embodiments, the visual attention robustness assessmentsystem 402 is easy enough to use that a person unfamiliar with visualsaliency theory could use the system to evaluate the robustness of anobject, region, or scene. User 401 may be a consultant hired by, or anemployee who works for a commercial entity to evaluate positioning ofsigns and non-sign objects within their environment. User 401 may alsobe a content designer interested in evaluating the design and placementof a digital object (such as advertisement) on a web page in which thevisual properties of the page may change.

FIG. 8 shows an alternative embodiment of the visual attentionrobustness assessment system, in which the visual attention robustnessassessment system additionally includes web server module 501. Webserver module 501 is shown, for convenience, as part of visual attentionrobustness assessment system 402. Web server module may, however, beembodied as a software module running on computer system 408, inseparate memory space. Or web server module 501 may be on a separatecomputer system coupled, via network, to the visual attention robustnessassessment system 402.

Web server module 501 provides an interface by which user 401 maycommunicate with visual attention robustness assessment system 402 viaclient computer 503 and via network 502. In one configuration, webmodule 501 executes web server software, such as Internet InformationServer from Microsoft Corporation, of Redmond, Wash. Web server module501 provides a mechanism for interacting with remote user 401 throughthe use of, for example, Active Server Pages, web pages written inhypertext markup language (HTML) or dynamic HTML, Active X modules,Lotus scripts, Java scripts, Java Applets, Distributed Component ObjectModules (DCOM) and the like.

Although illustrated as “server side” software modules executing withinan operating environment provided by computer system 408, the functionalmodules that comprise the visual attention robustness assessment system402 could readily be implemented as “client-side” software modulesexecuting on computing devices, such as client computer 503, as used byuser 401. Visual attention robustness assessment system 402 could, forexample, be implemented as Active X modules executed by a web browserexecuting on client computer 503.

Network 502 may be any kind of network, public or private. In oneembodiment network 502 is the Internet.

User 401 in FIGS. 7 and 8 may be any user of the Internet. In oneembodiment, user 401 may have pre-arranged, via a license, to accessfunctionality of the visual attention robustness assessment system.

FIG. 9 is a high-level flowchart illustrating one embodiment of theprocess the visual attention robustness assessment system 402 may employto evaluate the robustness of an object within a scene or a sceneitself. First, the visual attention module is invoked and provided sceneinput (step 601). Predicted objects from within the scene are thenreceived from the visual attention module (step 602). Next, someindication of the locations/objects/regions that the model predicts tohave high relative saliency is saved to a database (step 603). Preciselywhat will be stored in the database depends largely on the type ofoutput received from the visual attention module in step 602. Thedatabase, in one embodiment, stores information about the objects thatreached a threshold saliency value. In another embodiment, the databasestores a matrix of values, one for each object identified in the image,and stores, for example, the number of times that the object reached athreshold saliency value. Next, a check is made to see if the process iscomplete (step 605). In one embodiment, this check might determine ifthe current iteration exceeds the number of iterations initially set byuser 401. In another embodiment, the number of iterations may bedetermined by the algorithm. In yet another embodiment, the number ofiterations may be determined by properties of the scene or statisticsconcerning the robustness measure. For example, if one were attemptingto determine which of two advertisements would be better at a particularsite, one could run the algorithm until there was a statisticallyreliable effect between the two advertisement's robustness values.

If robustness evaluation is not complete (“no” at step 605), IE changeswill be introduced into either the visual attention module's visualattention model, or the scene input. IE changes may be divided into twocategories: structured and random. Random variability is variabilitythat is uncorrelated. For example, random variability in a scene mightinclude random variations in the individual pixel colors and/orintensities. In such case, the pixel variations are uncorrelated. Incontrast, structured variability has correlation between the elementsbeing modified. For example, a scene modified by simulating the movementof an object within the scene, or adding or removing an object within ascene would constitute structured scene variability. In such case, thechanges in the pixel modifications are correlated. Random internalvariations might include random variations in the attention model usedby the visual attention module. Structured variability, on the otherhand, might be programmatic biases for attention in one part of theattention map over another. The types methods of generating variabilityare summarized in Table 1. The sub-processes that introduce IE changesare further detailed below.

TABLE 1 Internal External Structured There are many ways to Variabilityin the image in generate Internal Structured which collection of pixelsco- variability. This type of vary. This co-variation can be variabilityis defined by co- generated by defining a region variation over maps orspatial (or regions) within the image regions. One method would andmodifying the RGB be to vary the weighting of values within such region.All the different feature maps to of the pixels within a the saliencymap. The particular region would be weighting values may vary modifiedto the same degree. slightly from one trial to the The amount ofvariation from next, simulating an individual region-to-region, ortrial-to- preferences or expectations of trial may be set by the color,orientation, luminance, standard deviation of a etc. Another method isto Gaussian distribution. A have the model bias one region of thesaliency map second parameter might be the (for example, lower “size” ofthe region that can hemisphere) in which all of co-vary. the locationsare given a saliency ‘boost’. Random Random variation is added toPixel-by-pixel variation in the output of the saliency which the RGBvalues are map before the visual randomly modified. The attention modeldetermines degree of modification may be where attention will bespecified by the standard allocated. The variability deviation of aGaussian could be in the form of any distribution in which the meanprobability distribution, but of the Gaussian is set to the the oneexample practice is to “true” RGB value. Other use a Gaussiandistribution in methods for variability can which the mean of the alsobe used. Gaussian is set at the expected saliency value and the width ofthe Gaussian is set by an appropriate standard deviation.

Once the IE changes have been made, the process returns to step 601.

Once the process is complete (“yes” at step 605), analysis is done onhow, and the degree to which, the objects predicted by the visualattention module changed in successive iterations (step 606). Thisanalysis is further described below.

Finally, output indicative of robustness is generated (step 607). In oneembodiment, this output is provided to a user via graphical userinterface module 406. However, the output may also be provided in otherforms to, for example, other programs or calling functions.

Changes to the Visual Attention Model

One example of modifying the visual attention model (a type of changethat could be introduced in step 604 in FIG. 9, discussed above) is toprogrammatically introduce a bias toward a particular feature within thescene, or a particular area within the scene. Such variability to thevisual attention model will have an effect on where attention will beallocated and will simulate variability between observers and/orvariability between human observers experiencing a scene from one timeto another. As an example, one might use the Itti and Koch bottom-upvisual saliency model to evaluate visual attention robustness of ascene. With such a model, a saliency “map” is generated by combining theinput from the different visual feature maps. For example, in someinstantiations there are three feature maps: a first that is sensitiveto color, another to orientation and a third associated with luminance.The input to the saliency map is a weighted combination of these threemaps. Typically, these three maps have equal weights into the saliencymap indicating that there is no bias for one type of feature overanother and can be represented as a weighting vector (for example, [1 11] for an equal weight of the three feature maps). One method forgenerating variability in the observer's biases is to modify theseweights simulating a viewer that might be biased toward one feature overanother. For example, one could bias the model toward brighter items inthe environment by setting the weights to [0.5 0.5 2]. One method ofgenerating internal variability (or simulating viewer variability) is togenerate a collection of models that have different weighting values.This could be done by randomly setting the values for each model.

Another approach is to randomly modify the model's internalrepresentation of the scene. For example, the neurons in the humanvisual system are somewhat noisy in that their activation will vary evenwith the presentation of the same image. Simulating this variability canbe done by perturbing the internal representation of the valuesassociated with the individual feature maps (for example, color, shape,orientation, brightness) a visual attention model develops in responseto a scene.

Another method is to perturb the values in the saliency map that is usedto identify which regions of the image are most salient. For example,one could perturb the internal representation following each fixationcalculation, or perturb the values independently each time a scene ispresented.

Changes to the Scene

The scene is changed by creating a new image using a graphical editor(e.g., Photoshop), 3D representation of a scene (for example, virtualreality model), or video (again, all collectively called “scene”herein), based on the previously existing scene in which the propertiesof the scene are varied. For example such varied properties couldinclude simulated (or real) changes in lighting, or newly added orremoved objects (real or virtual), or varying the pixel colors randomly.

There are many different ways to change a scene. Deciding what externalchanges to make can vary from situation to situation. One method fordeciding what types of changes to make to a scene is to determine thetypes of scene variability that typically occur within the scene andintroduce those changes to the images that are analyzed. For example, ina hotel lobby people will be moving through the scene. Sometimes theguests will be standing at the counter, sometimes standing at theelevator, or walking from one place to another. Simulating thesesituations provides a method for capturing the variability in the actualscene.

One may want to vary and/or measure the degree to which the externalvariability is added to the scene. One method for quantifying the degreeof external variability is to vary the number of objects manipulated inthe scene. For example, returning to the hotel lobby example withpedestrians, one may vary the number of pedestrians in the scene. Fewpedestrians would translate to a small amount of variability while alarge number of pedestrians would translate to large variability.Another method for measuring variability is to measure the pixelvariation for all of the images that are generated. One example measureof pixel variability is to measure the average variance of each pixelfor the set of images that are generated. Once the variability of eachpixel is calculated, a single number could be produced by computing themean variance of all of the pixels. Again, as the variability of thewithin the image increases, this value will also increase.

One method for generating changes is to make changes to an image bydigitally modifying the image using commercially available image editingsoftware such as that marketed by Adobe Systems Inc. of San Jose underthe trade name “Photoshop.” Using this method one can modify the sceneby digitally placing objects, removing objects, or simulating lightingchanges. These changes would then be translated into a collection ofimages that would be read and analyzed by the model. These changes canalso be applied to images automatically by superimposing layers ontoimages algorithmically.

Another method for generating external structural variability is togenerate virtual representations of the scene. Using this approach onecan easily modify the locations of objects, object properties, andlighting conditions. From these virtual representations, images can begenerated in which the objects, their properties, their poses, andlighting conditions have been changed.

Thus far, approaches to changing the scene could be said to be“synthetic”—that is, they are changes to representations of the scene,rather than to the scene itself. Another approach to modifying the sceneis to modify the actual scene. For example, one could take a series oftime-delayed images (for example, a picture every 300 seconds for adefined period). This series of pictures could then be used forsuccessive iterations of the visual attention robustness assessmentsystem. Such analysis (multi-image across time) would provide a measureof the external structure variability. Furthermore, given suchvariability (people and objects moving, along with changes in lightingdirection due to changes in, for example, the sun's position) one couldanalyze where attention will typically be allocated within theenvironment.

In addition to adding external and internal variability “blindly” onecould also introduce variability that is dependent upon the visualsaliency of the different objects in the scene. For example, one maywant to decrease the degree of variability where attention is allocatedand increase it for those regions where attention is not allocated.

Evaluating Robustness

There are a number of methods for characterizing the robustness of ascene, such as using a “heat map” graphical representation, an internalentropy value, a robustness index, or robustness value. One basicapproach to summarizing the integrated results from repeatedly applyinga visual attention model to a scene, while for each iteration applyingsome type of IE change, is to generate a graphical representation of therelevancy values or regions that are overlaid onto a representation ofthe scene. Such an approach yields something akin to a heat map, with“brighter” objects being more robust to the variability in question.

In addition to evaluating the effect of noise on which objects a visualattention model predicts, another method for evaluating robustness is tomeasure the variability in the model's internal representation of thescene. For example, the Itti & Koch model uses an internal “saliencymap” representation to determine the locations that visual attentionwill be allocated. One could (but not the only method) measure theamount of change, or entropy, in this internal representation of thesaliency map as a function of IE changes, as a metric for robustness.One could also look at the responses of the individual feature maps, orother internal representation of a visual attention model to measure theeffects of IE changes.

Another method for generating a robustness metric is when the IE changescan be parameterized. This would be the case, for example, where oneadds random luminance noise to the scene (thus changing the amount ofnoise by modifying the luminance energy of the noise relative to theluminance energy in the image). Or one could increase the variability ofthe connection weights of the visual attention model and characterizethe degree of change as a measure of entropy. Another method forcharacterizing the robustness of an object, group of objects, or sceneis to determine the amount of noise that these objects can toleratewhile retaining their visual saliency hierarchy. The method for thismetric is to measure the likelihood that an object will be attended as afunction of the amount of variability generated. Those objects and/orscenes that are highly robust will tolerate large amounts of noisebefore attention begins to be attracted to the noise elements and not tothe objects/regions. The resulting metric would be the level of noisethat the object could tolerate before attention was drawn away from theobject and toward the noise.

FIG. 4 is a graph showing the probability of attending to an objectversus the degree of variability for two objects, the first object 451having a low noise tolerance (low robustness value) and a second object452 having high noise tolerance (high robustness value). The graph showsthe effect of variability on whether attention will be allocated toobject 451 and object 452, each with different respective robustnessindexes. In this example, adding a small amount of variabilitysignificantly affects the probability that the model will predict thatattention will be allocated to Object 451. This is illustrated in FIG. 4by the drop in probability with only a small amount of variability. Bycontrast, Object 452 is unaffected by small amounts of variability. Thisis illustrated by a “Drop-off” point occurring with a much greater levelof variability. By measuring these “drop-off” points one can quantify,in terms of a robustness index, different objects within a scene.

Rather than, or in addition to, the heat map approach just described, itmay be useful, at times, to quantify the robustness of a particularobject (or plurality of objects) within a scene. Such quantification mayallow for subsequent analysis such as determining the degree (or therobustness index value) of IE changes that begin to reduce the degree ofsaliency of a particular object within a scene.

It may also at times be useful to evaluate and/or quantify therobustness of the scene itself. This distinction between object andscene robustness opens the door to many types of interesting subsequentanalysis. For example, it is possible that a scene has a low robustnessvalue (that is, the set of objects that the model predicts will receivevisual attention vary as a function of IE changes) while an objectwithin that scene has a high robustness value. An example of a “high”object robustness value and a “low” scene robustness value would be, forexample, when the target object is always in the top three attentionfixations, but the other objects being considered vary significantly.

Table 2 provides exemplary data in which a target object has a high andlow robustness value crossed with scenes that have low and highrobustness values. In this table, one possible robustness metric isshown in which a normalized likelihood value, called a Robustness Valuebelow, is calculated for each object. For each scene the left columnindicates the percentage of time that the object appeared in the firstthree saccades when there are IE changes. In the example shown in Table2, there are 14 objects within the scene, one target object and 13distractors, distractors being non-target objects or regions havingsaliency. If the model were to choose three objects randomly, eachobject would have a 21.42% chance of being selected (3× 1/14=0.2142).This would be the condition in which there is very little object and/orscene robustness, and thus it serves as an important baseline by whichto compare the robustness of the scene. This is because robustnessdeclines as probability approaches that of chance. In this example, theRobustness Value is mathematically the % Attended divided by the %chance of being selected, thus the first Robustness Value of4.6667=100%/21.42%. One method for calculating a robustness value forthe scene is to calculate the average robustness value for the top Kobjects (number of attention selections). At the bottom Table 2 is theaverage robustness value for the top 3 objects.

TABLE 2 Robustness Target Target Target Target Object = High Object =High Object = Low Object = Low Scene = Low Scene = High Scene = HighScene = Low Robustness Robust Robust Robust Object ID Attended ValueAttended Value Attended Value Attended Value Target Object 100% 4.6667100% 4.6667 5% 0.2333 9% 0.4200 Distractor_1 15% 0.7000 90% 4.2000 97%4.5267 30% 1.4000 Distractor_2 15% 0.7000 5% 0.2333 5% 0.2333 26% 1.2133Distractor_3 17% 0.7933 2% 0.0933 2% 0.0933 15% 0.7000 Distractor_4 14%0.6533 80% 3.7333 88% 4.1067 32% 1.4933 Distractor_5 13% 0.6067 3%0.1400 91% 4.2467 28% 1.3067 Distractor_6 21% 0.9800 2% 0.0933 2% 0.093322% 1.0267 Distractor_7 18% 0.8400 1% 0.0467 1% 0.0467 32% 1.4933Distractor_8 19% 0.8867 4% 0.1867 4% 0.1867 33% 1.5400 Distractor_9 16%0.7467 2% 0.0933 2% 0.0933 38% 1.7733 Distractor_10 20% 0.9333 1% 0.04671% 0.0467 18% 0.8400 Distractor_11 14% 0.6533 2% 0.0933 2% 0.0933 17%0.7933 Distractor_12 18% 0.8400 8% 0.3733 8% 0.3733 9% 0.4200Distractor_13 19% 0.8867 0% 0.0000 0% 0.0000 30% 1.4000 Scene Robustness2.1933 4.2000 4.2933 1.6022 Value

Table 2 categorizes the locations where attention is allocated, byobjects. But as mentioned before, the term object is loosely defined asa region or area in an image (or scene). The methods and systemsdescribed herein are not limited, however, to an object-basedapproach—other, similar approaches would also work. For example, a gridcould of equal size regions could be defined over the scene, or regionsof the scene defined based upon properties of the human visual system(for example, the size of the fovea of the viewer).

Assessing Across Scene Robustness

Given the above methods for evaluating the robustness of an object or ascene, it is next possible to extend the evaluation of robustness inother ways. For example, “across scene robustness” is a measure of howthe robustness of an object (or a group of objects) changes acrossdifferent scenes. By providing a robustness metric for an object (asdiscussed above), separate from the scene, one can evaluate therobustness of a particular object in different scenes. For instance,consider the situation in which an advertiser is deciding between twodifferent pieces of advertisement content that will be placed on threedifferent billboard signs that are in three different scenes. FIG. 5provides an illustration of an example of this scenario. Scene 703includes billboard 700. Billboard 700 repeats in scene 704 and 705.Similarly, billboard 706 is seen in scenes 707, 708, and 709.

Using methods described earlier (for example superimposing theadvertisement content in a graphical representation of the scene) onecan determine a robustness value for each billboard in each of the threescenes—generating six different object robustness values. The customermay then select the advertisement (object) that has the highest averagerobustness value (or other metric based upon the collection ofrobustness values).

One can also use similar analysis when one has a set of objects butneeds to select from a set of scenes in which to place the object orobjects. If we continue with the billboard advertising example, thecustomer may have a single advertisement that they want to place in twoof three different scenes. One can employ the techniques described aboveto generate the digital images that can be used to generate therobustness value for each advertisement in the three different scenes.To facilitate the scene selection, the customer could choose the twoscenes (with the advertisement superimposed within the scene) that havethe highest robustness values.

In addition to selecting a single object (for example, advertisementbillboard) from a collection of objects for a fixed set of scenes, or aset of scenes (from a collection of scenes) for a single object, onecould also assign objects to specific scenes based upon a robustnessvalue. For example, a customer might have three different advertisements(the content that is placed on a billboard) that they have developed andtwenty different billboards that these signs could be placed upon. Bygenerating robustness values for each of the three advertisements placedin the twenty scenes, the customer could then choose to use theadvertisement that has the highest robustness value for each individualscene. Additionally, one could assign the best advertisements to the tenscenes with the highest robustness value.

Although the examples thus far illustrate how a robustness metric couldbe used to improve choices made with regard to billboards andadvertisements, the benefits of a robustness metric are not limited tothis domain, and could be used in any domain in which one has acollection of contexts (scenes) and a collection of visual stimuli(objects) to select between, such as digital signs within a store orhotel, static signs, product signs, product packaging configuration, orweb sites.

Note that the systems and methods described herein for evaluating therobustness of objects within a scene, or the robustness of a sceneitself, are not dependant on any particular methodology for determiningvisual attention. Rather, they can generally be used with any model forassessing visual attention, and in some embodiments multiple differentvisual attention models may be used for evaluating robustness of anobject or a scene.

Robustness and Attentional Fixation Sequence

The discussion up to this point has mostly focused on the robustness ofan object in a scene or the robustness of the scene or a collection ofobjects within the scene without consideration of the order in which theobjects are attended to. For example, analyses up until this point haveevaluated only whether or not the model actually predicted thatattention was allocated to the object or not. There are, however,situations in which the order of the elements actually does matter. Forexample, for a multi-object scene, one may want to know how often oneobject appears before another object, and whether there are interveningobjects that are attended to in between the attention to two separateobjects (so-called intervening objects).

Using the methods described above related to introducing variability toa scene one will have available to them a collection of attentionfixation sequences. Using this data one can identify all of thesequences in which the target order (for example, attending to Object-Abefore Object-B) is achieved. A sequence robustness analysis could usethe probability of achieving the target sequence as a measure ofsequence target robustness. One example method for measuring sequencetarget robustness is to compute the likelihood that the target order(that is, Object-A before Object-B) would occur if one were to randomlyselect the objects. The sequence target robustness value would be theprobability that the target sequence was achieved divided by thelikelihood that it occurred by chance. The principles behind this metricare similar to the principles behind the robustness value describedabove.

Considering a first case where the relative order of attention doesmatter, but intervening objects do not. The objective is to haveattention drawn to Object-A before Object-B, as would be the case forexample where a company is advertising a dinner special for a restauranton a billboard near the restaurant itself. The company may beconsidering two discreet advertisements the dinner special. The goal isto get people who are passing by to first attend to the dinner specialadvertisement followed by attending to the restaurant itself. Using themethods described above to evaluate the two different advertisements(that is, digitally inserting the advertisements in a digital image sothey appear as if on the billboard), as positioned within the scenesthey will be placed) then applying IE changes, one can calculate howoften the billboard receives visual attention before the restaurant.Tables 3 and 4 provide possible likelihoods showing the relativeordering of Object-A and Object-B along with whether they receivedattention at all (Object-# Not Att.). From these tables we see thatObject-A before Object-B occurred 65% of the time with AdvertisementContent-1 but only 40% with advertisement Content-2. Thus, if thissequence was important the customer might be inclined to chooseAdvertisement Content-1.

TABLE 3 Advertisement Content-1 Object-A Object-B Object-A Pos 1 Pos. 2Not Att. Object-B Pos. 1 25% 7% 2. Object-B Pos 2. 65% Object-B Not Att.2% 1%

TABLE 4 Advertisement Content-2 Object-A Object-B Object-A Pos 1 Pos. 2Not Att. Object-B Pos. 1 20% 20% 2. Object-B Pos 2. 40% Object-B NotAtt. 5% 15%Robustness and Multi-Sample Conditions

The discussions thus far have focused when observing an object from asingle location. However, the world is highly dynamic, and often timesthe observer is moving through space. As the observer is moving throughspace, the observer can have multiple chances to “process” a particularobject. However, this movement generates another important aspect to therobustness analysis which is the amount of time, or the number offixations, in which an object will be visible, can vary. We will referto this as the visibility duration—how long a particular scene or objectis visible by an observer. One method for capturing the visibilityduration is by using a model that reports a predicted sequence of visualattention locations. With these models the visibility duration can bespecified by limiting the number of fixations that the model considersfor the robustness evaluation for the scene (this also applies to sceneoptimization, discussed below). Longer visibility durations correspondwith more fixations, while shorter visibility durations would correspondto fewer fixations. Changes in the visibility duration can have asignificant effect on the robustness analysis.

When the number of fixations that can process a scene is limited,whether an object will receive attention or not will depend on where itis in the visual attentional sequence. For example, a customer may placea digital sign within a hotel. The digital sign is presenting two piecesof content—one after the other. One piece of content is presented for 3seconds and the second for 6 seconds. Given its limited visibilityduration the 3-second piece of content needs to appear earlier in theattentional sequence than the 6-second piece of content. If for example,people make 2 fixations per second, the model has to predict thatattention will be drawn to the object in the first six fixations. Bycontrast, the 6-second piece of content has to receive visual attentionin the first 12 fixations. Given these dynamics, and others like them,not considering visibility duration can lead to inaccurate predictionsabout the objects that people will attend to in a scene.

To illustrate this, we will extend the billboard example (but this sameconcept applies to any display that is dynamically changing). Let usconsider a long straight road in which there are multiple signs alongthe road. Furthermore, the road is flat and there are no other objectsother than the billboards. Under these conditions all of the billboardswould have the same visibility duration. That is, the amount of timethat the visual system processes any particular billboard is the sameand will be determined by the speed of the car, the size of the sign andany atmospheric aberrations in the scene.

Let us now consider this same road in which there is a row of trees 500feet in front of one of the last billboard on the road. All of the otherbillboards would have equal visibility durations, but this finalbillboard would have shorter visibility duration. Given this change inthe visibility duration one would want to limit the number of fixationsthat the model would consider for the robustness evaluation. The firstset of billboards might have a high number of fixations, while whenconsidering the duration for the final billboard one might considerfewer frames from a movie or fewer fixations in an image.

This aspect of visibility duration can play an important role when oneconsiders the situation where there are multiple objects within a scene.Under some conditions one may want to distribute the attention capturingelements such that the objects are salient enough to be processed (orcapture attention) but not so salient that they distract from otherobjects within the scene. (Note: this concept also relates to sceneoptimization, which concerns modifying a scene in a way that achievesparticular goals. Scene optimization is discussed further below.) Giventhis, one would want to consider the visibility duration when analyzingthe robustness of an object, a collection of objects or a scene. Forexample, for an object that has a short visibility duration on a path,one may want to increase the saliency elements (for example, motion,brightness, color contrast, etc.) to increase the likelihood that aparticular object will be processed during that short visibilityduration. However, for another object that is visible along that samepath, but has a longer visibility duration, one may be able to takeadvantage of the fact that there are more opportunities to process thatobject and thus one can allocate fewer (or weaker) saliency elements tothat object.

The visibility duration would be factored into the robustness analysisby considering whether the object received visual attention anywherealong the path. Thus particular objects that have longer visibilitydurations than others would have more attentional fixations associatedwith them, and if attention was allocated to the object anywhere alongthat sequence then it would be considered a “hit.” Therefore, objectswith longer visibility durations would have more samples and would havea higher likelihood of being processed and thus may require a lowerlevel of saliency to be processed. An object with shorter visibilityduration would have fewer samples and thus would be less likely to beattended and thus, may require higher level of saliency elements to bedetected during that shorter sequence.

Scene Optimization

Up until this point, the disclosure has focused on robustness. Now weturn to other visual attention modeling related concepts that, in someembodiments, may benefit from the robustness-related methods and systemsdescribed earlier, but do not necessarily require assessment ofrobustness. One such related concept is that of scene optimization.

As discussed earlier, visual attention models exist that may predictwhere a subject will allocate his or her visual attention within ascene. However, such models do not provide any mechanism for identifyinghow a scene can be modified to achieve a specific visual goal. Because ahuman's visual system does not actually process the entire viewable areaof a scene, but instead only processes those regions in which attentionis drawn, it is desirable in many real-world situations not only to getpeople to ‘view’ specific objects within a scene, but to have them‘attend’ to specific objects.

A visual goal, then, refers to the desired manner in which a subjectwill attend to objects within a scene. For example, a visual goal couldbe simply a desire that particular objects are attended to (that is, thecollection of objects within a scene that one determines as important(from a visual attention perspective) in addition to the collection ofobjects that are deemed unimportant or even detrimental. Or, it could bea desire to have particular objects attended to in a particular sequenceor at a particular time, or it could be a desire to have particularobjects attended to from a particular viewing point, but not necessarilyfrom others. The remainder of this section discusses ways in which onecould utilize a computational model of visual attention to optimize ascene in order to achieve a visual goal. Some of the methods that willbe discussed utilize an assessment or robustness, as discussed earlier,but some do not.

Referring to FIG. 10, once a visual goal has been defined (step 1201),scene optimization starts by assigning an explicit cost/reward structureon the objects within the scene (step 1205). Such assignment defines,quantitatively, the visual goal. The cost/reward structure defines whichobjects within the scene are high value objects, low value objects andeven objects that the user views as distracting or detrimental to thevisual goal. The user will place “attention utility” values on to eachof the objects that are deemed to be part of the visual goal (positiverewards) or are detrimental (negative costs). Or, the user can placepriorities specifying which elements are “more” valuable to the visualgoal than others.

Next, a number of possible changes to the scene are defined (step 1210).These could be simple changes such as lighting, color of objects,positioning of objects, etc. or more complex design changes such aswhere within a lobby a reception area should be built. Of course,ideally, the time to evaluate something as fundamental as positioning ofthe reception area is best done before building a hotel lobby, so it isexpected one utility of the scene optimization methods described hereinwill be for evaluating synthetic, or partially synthetic, scenes anddesign/layout choices within such scenes.

The actual real-world “cost” (or estimate) is then associated with eachpossible scene change (step 1215). For example, in the case of a hotellobby, where the goal is for patrons to attend to a particular sign,changing the color of the sign may be relatively inexpensive (could beassigned an estimated change value of $200), while changing the color ofthe granite floor would be expensive (could be assigned an estimatedchange value of $20,000). Assigning real-world cost estimates makes itpossible to associate a price figure with a plurality of changes. Forexample, a range of possible changes to a scene could be considered,some of which meet all goals, some of which meet most goals, some ofwhich meet goals and do so the most inexpensively, and others of whichmeet 90% of the defined goals, and to achieve the other 10% of goals, itmay be found it takes a disproportionate investment capital. In otherwords, associating real-world costs with possible changes may, in someembodiments, allow for a much more useful assessment of options. In theend, the method provides the scene configuration that maximizes theattentional utility while minimizing the cost associated with the objectfeature configurations (step 1220).

Note that real-world costs are just an example of how relative costs ofparticular changes could be co-associated—other systems, such as pointsystems with higher points correlating with higher costs for particularchanges, and lower points being less expensive changes, could just aseasily be used.

A graphical editor with the ability to track and manipulate discreteobjects may be useful in defining possible changes to a scene. Forexample, a sign in a scene that is being viewed within such a graphicaleditor could be identified (right-clicked with a mouse, for example),and the user would be able to select the changeable properties of thatobject. These changeable properties might include color, lighting,positioning within layers (for example, the object could be put in frontof or behind some other object), contrast, shadow, size, etc. Inaddition to selecting the individual properties that may be changed, theuser may also have the ability to define the allowed scope of change orother relevant parameters. For example, with respect to color, the onlycolors a client or user may find acceptable for a particular wall thatis within a scene may be shades of tan. Thus the color attribute isdefined to only be varied within the specified spectral range.Similarly, if the attribute is size, there may be an obvious limit tothe size a particular object may grow to (or be reduced to) before thesize of the object is not acceptable to the client or user. As mentionedearlier, the real-world cost estimate is associated with each possiblechange. Where there is a range of possible changes, a user may definethe cost of exemplary points within the range and the supportingcomputer system (later described) will extrapolate a best-fit curve forthe example points. The user may be presented with a number of possiblebest-fit algorithms and thus be asked to select which one to use. Forexample, the user may just want to define, with respect to the size of asign, that the smallest allowable sign costs $100, and the cost of theintermediate signs increase linearly (based on size) between these twocost points.

With goal (or plurality of goals) defined, the attentional utility forthe goals, the possible scene changes, and the costs of the possiblescene changes defined, the next step is to evaluate the benefit ofpossible changes to the scene and attempt to find the sceneconfigurations that achieve the visual goal defined in step 1201. Thisis done, in one embodiment, by algorithmically modifying properties ofthe scene to maximize the expected reward, or the scene configurationthat minimizes costs while maximizing rewards, or the sceneconfiguration that simply meets the defined goals for the least cost.The expected reward is calculated as the likelihood that a particularobject will receive attention and the reward/cost for attending to thoseobjects. In addition to the reward/cost for attending to the location inthe scene, the optimization routine also factors in the cost/reward formaking specific types of changes to the scene (for example, changing thecolor and/or position of an object within the scene). One method forcalculating the likelihood of an object receiving attention may bedetermined using the robustness analysis described in the previoussection.

Equation 1 provides one example of how to formulate a utility functionusing a computational model of visual attention.

$\begin{matrix}{{{ER}(F)} = {{\sum\limits_{a_{F} \in A_{F}}\left\lbrack {{a_{F}({xy})}{R_{a}\left( {O_{f}({xy})} \right)}} \right\rbrack} + {\sum\limits_{o \in O}{R\left( o_{f} \right)}}}} & {{Equation}\mspace{14mu} 1}\end{matrix}$

F is the set of changeable attributes for the objects within the scene.For example, it would include the color, texture, or position for eachof the objects within the scene. A is the set of attentional fixationsthat the model predicts given this feature configuration F. R(Oxy) isthe reward (both positive and negative) for attending to the object atthe position (xy) with the feature set f. a(xy) is the likelihood thatthe model predicts that attention will be allocated to location xy. Forsome models and approaches of visual attention a(xy) can be a binaryvalue (0 or 1 indicating whether attention will or won't be allocated tothe location), while for other models and approaches, this might be alikelihood that attention will be allocated to this location (0 . . .1). Either way, this part of the equation specifies the reward for theallocation of attention for the current object feature set.

The other part of the function specifies the cost for using feature setf for object o. R(O_(f)) specifies the reward/cost for using feature setf on object o. In some cases the cost for a particular feature might be0.0 (for example, this might be the case for the current feature set forobject o). However, one may want the model to consider all possiblefeature combinations being equally difficult. In this case the rewardsfor all features would be equivalent (or most easily 0.0). However, insome situations (for example, changing the color of the carpet in ascene versus moving a vase) there will be constraints on the allowablefeature configurations. In such situations, one could specify thesevalues in terms of costs/rewards on the object (o) and the features (f),or they could simply refrain from defining the non-allowable featureconfigurations as an allowable change.

Using this reward function, the solution space is explored for a featureconfiguration that optimizes the stated reward function. There are anumber of methods for achieving the optimal solution once the rewardfunctions are specified. These methods include, but are not limited to,using closed form equations, Monte Carlo Simulations, SimulatedAnnealing, Genetic Algorithms, and Stochastic Gradient Descent. Inaddition to these approximation approaches, for some visual attentionmodels, one could implement a closed form analysis.

The solutions from the solution space that meet the goals, andassociated cost information, is then available to be evaluated.

Note that many of the examples thus far focus on optimizing a scene byincreasing the visual saliency of objects within that scene. Note,however, that some visual goals may be achieved by reducing the visualattention (or distraction) from other objects. The scene optimizationmethod described herein, depending on how the cost model has been setup, may result in muting aspects of the scene (not always making theobjects more visually salient).

Scene Optimization and Attention Sequence

As mentioned above, some goals may not consider the order in whichobjects are attended to. Under such conditions, the reward/cost forattending to an object will be unaffected by either its position in thesequence and/or the objects that were attended to before or after acurrent attentional fixation. However, there are situations in which theattention fixation order of objects within a scene may be important. Forexample, order tends to matter when a subject is following a specificsequence of instructions.

To deal with this more complicated visual goal, the expected rewardfunction above (Equation 1) may be expanded to optimize the featureconfiguration such that the expected reward is order dependent. This canbe done by taking advantage of a variable reward structure as a functionof saccade number. Note that in Equation 1 the reward for attending toan object is indexed by the individual attentional saccade (Ra). Byspecifying a different reward based upon the saccade's sequentialposition (a) one can generate a method by which the scene is optimizedby the predicted saccade order. FIG. 11 is a graph illustrating tworeward structures. One reward structure is represented by invariant withsaccade position (line 1225) and the second is dependent upon saccadeposition (line 1230). The saccade dependent reward structure specifiesthat the expected reward for attending to this particular object is veryhigh if it occurs early, but it declines as attention is allocated laterin the sequence. This type of reward structure might be associated with“high value” objects, such as pedestrians in a construction zone scene.

Rather than basing a reward on how early the object is attended in theattentional sequence (as FIG. 11 illustrates), one could also base thereward on a sequence-based goal, where an object's reward is based onthe objects that have received attention before it and after it. Forexample, a hotel owner may have two outdoor signs. One advertises aspecial in their restaurant and the second displaying the name and logoof their hotel. The hotel owner decides that it is important that theadvertising special sign should be seen before the hotel's sign. Giventhis visual goal the analysis would place a very high reward on when the“restaurant special” sign is attended to before the hotel name.Furthermore, a low reward would be given for when the hotel name isattended to before the “special advertising” sign and no reward may begiven if either one is attended without the other. The sequence could bea relative sequence (before versus after) or could be an absolutesequence (no reward for attending to objects A and B unless Object-Aoccurs as the first object that receives visual attention and Object-Bis the second object that receives visual attention). There are, ofcourse, many other ways in which reward structures could be formulated,as will be appreciated by a skilled artisan.

Thus far this section has discussed two possible reward functions inwhich the position of the saccade or the order of the saccades affectsthe reward. One skilled in the art will recognize one may define anyarbitrary sequential reward function over the set of objects and the setof saccades. More generally one can define this sequential component ofthe reward structure by defining an M dimensional space (one dimensionfor each object) that is the length of the saccade sequence in question.

The optimization routine may be set up to provide the best configurationfor a scene given the reward structure, but one may find that theresulting scene “recommendation” not appropriate. For example, one mightwonder how the attentional sequence changes if one were to modify thecolor of one object to make it more, or less, conspicuous. Making oneobject more conspicuous can have non-obvious, non-linear effects on thepredicted sequence. This is due to the fact that attention allocation isby definition a zero-sum game: allocating attention to one object willnecessarily mean that attention is allocated away from another object.Thus modifying the features of one object to increase its conspicuitywill not only change the likelihood that this object will receiveattention (and where in the attentional sequence it will receiveattention) but it will also affect the likelihood that other objectswill receive attention and where in the attentional sequence thoseobjects receive attention.

Because one approach to the optimization routine is to automaticallyconsider multiple feature configurations, many (or all) of the possibleconfigurations that define the solution space will have been explored.The non-optimal solutions could be of great interest to a user or aclient. For example, one might want to know what color to make a sign tomove its position from, say, the 25^(th) position in the saccadesequence to the top 5. The system could look through the storedattentional sequences in which the features of all the other objects areheld constant and the object of interest falls in the top 5 saccades.

Robustness, discussed earlier, can also be used in scene optimization.For example, one could optimize a scene not only to the properties ofthat image, but to the scene as it might appear at different times ofday, or to different viewers. That is, the optimization routinerecommends scenes that are robust to the variability that the scene mayexperience. Previously we described methods for capturing and generatingboth internal and external variability to the input scene and model. Thegoal of these approaches was to simulate (or capture) the expectedvariation of the actual scene. For example, one method for capturing thevariability of observers is to vary the parameters of the visualattention model (for example, the weights of the individual feature mapsfor calculating saliency) and running these models on the same image.Running each scene configuration through multiple instantiations ofdifferent model parameters (internal variability) would give multiplescores for a particular scene configuration—one attentional sequence foreach model. By taking the average score for each model configuration,one could generate an expected score for the scene configuration withthe given variability. One might recommend the scene configuration thatprovides the best score on average.

Also, one may define a visual goal in terms of robustness, and thenoptimize a scene's object (or objects) to particular robustness values.

Visibility Duration

As mentioned with respect to the robustness discussion, above, differentobjects or different scenes may have different visibility durations. Asa reminder, visibility duration refers to the period of time that anobject and/or scene will be viewed. Visibility durations, which may bespecified in terms of time will typically be translated into the numberof predicted attentional fixations (or salient regions) that will beconsidered in the optimization routine. The visibility duration would beused to limit the set of fixations that would be used for the analysisof the various scenes. More specifically, it will be used in the visualgoal analysis.

The visibility duration may have an effect when considering multiplevisual goals. As an example, one may have a document or content that hassix visual goals with different rewards—say the reward values are1,2,3,4,5,6. If the content is displayed on a digital sign for 5seconds, and people make about 2 fixations per second, this translatesto a visibility duration of 10 fixations. Given that the visibilityduration is 10 fixations, the visibility duration is long enough tocapture all of the visual goals. Under this condition, the model wouldmake the saliency of the six items relatively equal (assuming no otherobjects in the scene and an equal cost for making changes). The reasonthe optimization routine with visibility duration will make the visualgoals roughly equal, is that the model is more likely to attend to allof the targets under this condition. If the saliency of one of thetargets (say the object with the highest reward) is significantly higherthan that of one of the other objects (say the one with the lowestsaliency) then most visual attention models will attend to the mostsalient object first, then the second most salient, but eventually, itwill typically return to the most salient object again. If there is noadditional reward for returning to a previous object, this attentionalfixation will not increase the overall reward. However, by making theobjects relatively equal in saliency, the model is more likely to attendto all of the target objects and thus achieve more (if not all) of thevisual goals given the visibility duration. This aspect of theoptimization routine and the visibility duration is very different thansimply making the saliency of the object correlated with the targetobject's reward or relevancy.

FIG. 12 is a high-level diagram of scene optimization system 1255.Like-named components in this figure are similar in functionality andcapability to those modules discussed earlier. Scene optimization system1255 includes database graphical user interface 1245, which receivesinput from user 401, via computer 503, over network 502, to define oneor more visual goals for a scene, as well as possible changes to thescene. These possible changes to objects in the scene are stored indatabase 1235. Scene optimization module 1260 iterates explores theuniverse of possible scenes that meet the visual goal, and invokes scenemodification module 1240 to modify scene input in ways consistent withthe possible changes defined by user 401. This produces a modifiedscene, which is provided to visual attention module 403, which providesoutput relevant to visual attention, which is stored in database 1235.Scene optimization system 1255 may include web server module 1250 ifuser 401 is remote (and as shown in FIG. 12).

Multi-Perspective Scene Analysis

Up until this point the description has focused primarily on visualattention modeling utilizing a single, static image or a movie topredict where attention will be allocated within an image or images.These two approaches are often useful for many situations, but arelimited in that they operate on a single two-dimensional view of what isin reality a complex three-dimensional scene.

In the 3D world in which we live, small changes in perspective(orientation and/or position) can have significant changes on the imagethat is projected upon the observer's retina. Objects that are visiblefrom one perspective may not be visible at all in another. Furthermore,the spatial relationships (that is, the position of the projected imageof one object on the retina relative to a second object) between objectscan change significantly from one perspective to another. Becausechanges in perspective can generate large variations in the images thatare projected to the retina, they will also have significant effects onwhere human visual attention will be allocated within a scene. Forexample, FIG. 13A is an artist's rendering of a hotel lobby sceneincluding a reception desk having digital sign 1290 behind the desk. Thescene has been analyzed by a visual attention model which predicted thetwo objects that draw the most attention in FIG. 13A are the digitalsign 1290 and advertisement card 1285. FIG. 13B is the same receptionarea in the same hotel lobby, but digital sign 1290 is not among theseveral objects identified by the visual attention model. If the visualgoal is to have patrons attend to the digital sign 1290 and only thescene shown in FIG. 13A is analyzed, there will be a false sense ofsecurity that the visual goal is being consistently met. Thus there is aneed to have an effective way to measure and evaluate scene contentwithin a 3D scene.

Note that 3D scene evaluation (which herein we refer to asmulti-perspective scene analysis) is not the same as merely extending 2Dvisual attention analysis to such things as movie sequences. Such anapproach may provide data from many perspectives, but ultimately haslimitations similar to that of the single image approach, in that it mayprovide one with an ability to analyze the allocation of attention for aparticular movement through space, but it may not provide the ability toconsider other paths and/or movements through the space. Neither avisual attention analysis of static scenes nor a plurality of staticscenes in succession (videos) effectively deals with the fact that theimages are derived from a 3D setting.

So, then, it may be useful to have a multi-perspective visual attentionmodeling process that accommodates three dimensions and the myriad waysin which an observer may traverse a geographic space that is the threedimensions.

FIG. 15 is a flowchart showing the multi-perspective visual attentionmodeling process. The process begins with site planning (step 1329),which consists of determining which locations and objects within the 3Dscene one wants to analyze. In practice, this may mean acquiring ordeveloping a floor plan of the 3D scene that will be analyzed, anddetermining locations in the floor plan that will be representative ofthe observing perspectives of users. Of course, absent a floor plan ofthe 3D scene, a less rigorous approach could be used, in which the usersimply goes to the site and makes decisions about which locations are ofinterest and takes photographs from those locations. Additionally, theuser might record the position and orientation of where the image wastaken, which could be useful for reporting purposes.

Rather than photographs, video technology in which frames are capturedfrom the video or video sequences could also be used. When frames(images from video) or images (from digital cameras, or example) areused, the user can also use view interpolation techniques to generateviews that are between two different images. As mentioned earlier, theimages are not limited to being generated from a real environment, butthey can also be generated from synthetic (virtual) environments.However, in both cases, one must record or pre-specify the locations inthe environment where the images are taken from, and the perspective ofthe camera (which way the camera is pointed). A simple approach is tospecify locations, then have each successive image be generated from aperspective that advances 90 degrees around the north, east, south, westaxis (as is done in an example below). But absent pre-specifyinglocations and camera aiming protocol, the camera could instead betracked using GPS tracking technology possibly in combination withoptical tracking technology. For example, an instrument or instrumentswould be attached to the camera such that every time an image is taken,the system would record the three positional dimensions (X,Y,Z) and thethree orientation dimensions (yaw, pitch and roll) to provide anexplicit representation of the viewpoint from which the image iscaptured. These six values would be stored in memory and associated withthe image captured at that time. Of course, cameral position andorientation could be manually recorded.

With the 3D scene determined, next multiple images from the 3Denvironment are received, the multiple images representing a set ofviews that an observer may experience as the observer interacts andnavigates through a 3D environment (step 1330). In one embodiment, thisis accomplished by taking multiple photographs taken from multiplepositions and orientations within the 3D scene (again, such as the lobbyof a hotel). In another embodiment, a video is made of the 3D scene,with shots taken from multiple representative areas an observer might beexpected to be at. In yet another embodiment, a virtual 3D model is usedand views are generated by moving a virtual camera through the virtual3D model. No matter how generated, the result is a plurality of 2Dimages from various positions within the 3D scene. Data representativeof the location within the 3D environment and the camera orientation arealso collected, if not pre-specified. This data will allow one toevaluate a scene from many perspectives along with evaluating manydifferent paths that an observer might take through a 3D space.

Once images are collected, the regions of the two-dimensional image thatcorrespond to the objects of interest are selected and identified (step1335). This can be accomplished using a number of different methodswhich might include (but are not limited to) methods that automaticallyextract these objects, manual selection of regions, or even hybridapproaches that use both automated and manual tagging and labelingtechniques. An illustration of graphical results of one method (manual)for accomplishing this process is shown FIG. 14A and FIG. 14B. Theobjects such as mural 1310, digital sign 1315, and vase 1320 have allbeen identified by a manual selection process.

Next, the images including the objects of interest are processed using avisual attention model to produce visual attention data (step 1340). Asearlier mentioned, one such model is that described by Itti and Koch(2001) but any visual attention model could be used. The two-dimensionallocations where the model predicts that visual attention will beallocated when an observer is at each viewpoint are then recorded in,for example, a database. These locations are then compared to theregions that have been tagged and labeled for each viewpoint todetermine which objects within the visible area the model predicts willreceive attention.

After analyzing each individual viewpoint and computing which objectsthe model predicts will attract visual attention for each individualviewpoint, data concerning which objects will be attended from whichlocations is generated and stored. This data may then be used to, forexample, determine the likelihood that a particular object will beviewed (at all) by a potential observer as she traverses the 3D scene;the percentage of potential observers that will in fact observeparticular objects within the 3D scene; the likelihood of an objectbeing attended to for a sub-set of particular views (may be useful foranalyzing information about particular travel paths, such as entering alobby versus exiting the lobby), or the likelihood that an object willbe attended when that object is viewable (some objects may need toinconspicuous from most of the possible viewpoints within theenvironment, but for the viewpoints that the object is viewable, one maywant to have a high degree of certainty that attention will be drawn toit), or the viewpoints within a 3D scene that an object is visible (orthat the object will be attended to).

This 3D visual attention modeling may be combined with the systems andmethods discussed above with respect to scene optimization. The visualgoals that could be defined with respect to 3D visual attentionmodeling, however, may be different. For example, the visual goals maybe defined such that 90% of potential observers do in fact observe aparticular digital sign at some point while traversing a hotel lobby.

As an example of applying 3D visual attention modeling combined withscene optimization, consider a case where one may want an object toremain inconspicuous when observed from some viewpoint but beconspicuous from other viewpoints when the object becomes relevant. Forexample, a hotel that has three different forms of advertising contentthat it wants its customers to view and attend to during their visit. Afirst content is advertising a special on room upgrades; a secondcontent is advertising a special on room service; and a third content isadvertising tickets for a play that is being held in the ballroom of thehotel. These different forms of content are relevant at different timesduring the customer's visit. The room upgrade is relevant when thecustomer is checking into the hotel but is not relevant at other times.Room service is relevant when the customer is going to their room, butnot necessarily when the customer is leaving the hotel. The playtickets, in contrast, are relevant to customers at almost any time.Using the 3D visual attention modeling techniques combined with thescene optimization techniques (both described above) one can optimizethe placement and content of these advertising materials in thefollowing way. First, one can determine the locations in the hotel lobbywhere the information would be most relevant. For example, the roomupgrade is relevant when the customer is checking into the hotel, whichusually occurs when the customer is standing in front of the check-incounter. Areas next to an elevator might be best for general advertising(such as play tickets). Just as a hotel may want certain signsconspicuous from viewpoints corresponding to certain tasks (for examplecheck-in, or going to one's room), the hotel would also want signs thatare not relevant to be inconspicuous (for example, room upgrade whenwaiting for an elevator). One may analyze the position and content ofthe advertising material based upon the set of positions andorientations that one might be in when standing at the check-in counter.

To test one embodiment of the above-described 3D visual attentionmodeling and scene optimization method we took as our test 3D scene astandard conference room, a diagram of which may be seen in FIG. 16. Theconference room included a table 1345, a yellow sign 1350, green basket1360, telephone 1355, purple sign 1365, as well as other typical thingsone would expect to find in a conference room (chairs, waste basket,screen). Representative points from which an observer might be expectedto view the room were determined manually, yielding eight representativeobserving locations (observing locations 1366, 1377, 1378, 1379, 1380,1381, 1382, and 1383). For this test, the observing locations werespaced approximately 4 feet apart throughout the unencumbered regions(no furniture) of the room. Four images were taken, using a digitalcamera, from each of the eight observing locations, to yield 32 images.The arrows extending outward from the observing locations indicate thegeneral direction the digital camera was aimed for each picture—about a90-degree orientation difference for each picture at each observinglocation. The location and orientation for each of the 32 differentimages was recorded.

We then identified and tagged the pixels associated with 12 differentobjects that were found in at least one of the 32 images. This was doneby having a user select the polygon region that defined the 2D regionencompassing the object of interest on each of the 32 images. FIGS. 17 aand 17B shows an artist's rendering of two of the 32 images, wherepolygons encircle objects of interest, such as green basket 1360 andyellow sign 1350. “Tagging” simply refers to naming the regions thatcomprise objects (such as “yellow sign”). Note that the polygonsencircling the objects are representative of the tagging software'spurpose; the underlying images are not actually modified with thepolygon; the identified regions, once specified by the user, are notsignified in any way on the original image.

After tagging and labeling the images, the images were submitted to avisual attention model to collect the locations in the image where themodel predicts visual attention will be allocated. After submitting allof the images, the computer recorded, for each image, the x,ycoordinates where the model predicted that visual attention would beallocated. The computer also calculated for each of these attentionfixations whether it fell within a region of the image that was taggedand labeled by the user. The computer also recorded each image thatincluded a tagged region that was not predicted to receive attention (a“miss”). All data was stored in a database, which was then used togenerate a series of summaries concerning the conspicuity of the objectswithin the scene.

FIG. 18 shows graph 1395 illustrating three different example analysesthat were done for the 12 objects 1400 tagged and labeled within theconference room. The first analysis is the likelihood that the object isvisible or viewable (p(Visible)). This is the ratio of the images inwhich the object is in the image divided by the total number of images.p(Visible) is, then, a metric giving some indication of how well placedthe object of interest is within the setting. The metric that isdetermined is the likelihood that the object was attended given that itwas visible (p(Attended|Visible)), which was calculated by taking all ofthe images in which the object was visible and identifying whether thevisual attention model predicted that a fixation would occur in theregion defining the particular object. The metric calculated was thelikelihood that a particular object will be attended at all(p(Attended)), which is calculated by taking the number of images inwhich the model predicted attention allocation to an object at leastonce, then dividing that value by the total number of images.

Multi-perspective scene analysis represents the fact that a targetobject can be viewed from many different distances. For example,consider the billboard example described above. As one is driving down along flat road, there are many opportunities to attend to, and thusprocess the billboard. By contrast, for another billboard, there may bea hill or a group of trees that are occluding the sign until the verylast minute. By taking into account these different viewpoints, one canmore accurately analyze the likelihood that an object will receivevisual attention from the different perspectives that it can be viewed.Without multi-perspectives, and using only a single view, one mayincorrectly predict that an object will or will not receive visualattention given the number of perspectives that it could be attended.

FIG. 18 shows a subset of possible evaluations that one could carry outon data resulting from the analysis described above. There are a numberof conclusions one could draw from the data. First, note that thePurpleSign object is visible (white bar on graph) less often than theScreen object (that is, it was in fewer of the images taken within theenvironment). However, if we look at the black bars for these twoobjects, we see that when the PurpleSign is visible (that is, it ispresent in the image) it is always attended (p(Attended|Visible)=1.0),but when the screen is visible, is never attended. This suggests thatalthough the PurpleSign is located at a place where it won't be seenvery often, when it is in view the model predicts that attention will beallocated to it.

The PurpleSign object is, then, generating the types of properties thatwere discussed as desirable earlier in the context of the hotel lobby.That is, the sign is not visible from many locations (it isinconspicuous) but when people are in a location where it is visible (bythe check-in counter), they will almost always attend to that object.This is illustrated by the fact that the probability that the PurpleSignobject is visible (white bar) is about 15%. But the probability thatattention will be captured by the PurpleSign when it is visible (blackbar) is 1.0.

FIG. 19 is a block diagram showing high-level functional modules thatcomprise a multi-perspective scene analysis system 1415, which invarious embodiments may perform the multi-perspective visual attentionmodeling process described with respect to FIG. 15. It resides, in oneembodiment, in a computer system 408, which includes a number of otherfunctional modules (such as an operating system), and hardware, such asa memory or processor (neither of which are represented in FIG. 19).Though shown in FIG. 19 as a single computer, in practice variousportion of the functionality could be spread among several or manycomputers in a networked configuration. Multi-perspective scene analysissystem 1415 includes visual attention module 403 (discussed above). Italso includes multi-perspective scene analysis module 1425 whichinvokes, as needed, visual attention module 403 to do the analysis ofinterest (discussed earlier with respect to step 1340 of FIG. 15), andreceive results from the analysis (including which defined objects wereidentified by the visual attention module in which images), and storesthese results, or summaries of the results, in database 1410. Database1410 is any data storage device or system, such as a computer flat file,computer memory, or a database. Multi-perspective scene analysis system1415 also includes graphical user interface module 1420, whichfacilitates the input of multiple images (acquired in step 1330 in FIG.15), then, in this embodiment, facilitates the identification andtagging (step 1335 in FIG. 15) of objects of interest within the images.

User 1405 is any person or other computer system interested ininteracting with the multi-perspective scene analysis system. In oneembodiment, user 1405 is a consultant hired by a company to analyze andrecommend configuration changes to a 3D scene owned or controlled by thecompany.

Often times a viewer will remain in the same location for a reasonableperiod of time. For example, someone may be waiting in line to check outfrom a grocery store, a home improvement store or a hotel. During thistime the individual may engage in a “visual foraging” task. Visualforaging is a situation in which observer is not looking for anything inparticular, but simply looking around the environment for something thatis interesting. During this visual foraging the person will attend todifferent pieces of information by moving his eyes and when his eyesreach the edge of their rotation axis, the person will move his head.Typically he will move his head and eyes such that the item of interestis at the center of fixation. Current state-of-the-art does not simulatethis type of re-centering action on the images. Instead, when an imageis analyzed, the center of the image always remains fixed. This is trueeven when the fixation point is at the edge of the image (or visualfield). Without re-centering, the current state-of-the-art approach willonly be able to fixate to the edge of the image but will never fixatebeyond that point. Humans, by contrast, will attend to the edge of theirvisual field and rotate their head so their eyes are re-aligned to thecenter of gaze. This would allow the visual system to make anotherfixation in the same direction. With a single image, one cannot make afixation in the same direction due to the fact that there is no moreinformation off of the edge of the image.

The systems and methods described herein can, in some embodiments, usemultiple views of a scene or a single panoramic view to simulate there-centering of the eyes during visual foraging. This could be done asfollows:

-   -   1. Generating multiple images from a single viewpoint. The        images are generated by using a 360-degree panoramic camera or        multiple single images rotated around the viewing axis (vertical        or otherwise). Multiple images may be taken in which the views        “overlap” one another. The orientations of the views would also        be assigned to each view.    -   2. An initial view (“starting view”) is given to the visual        attention model. The view can be determined by the typical        viewing orientation that someone starts their visual foraging        (for example, in a store line, it might be looking “forward”        toward the cashier). One can also start with a randomly selected        orientation. In the case of a panoramic view, a “slice” of the        panoramic view could be used centered on the “starting” view. In        the case of multiple views/images, the image that is centered        closest on the starting location would be used.    -   3. The “starting view” is analyzed with a visual attention        model. The initial fixation is predicted by the model. The        orientation of this fixation would be calculated (this can be        done using trigonometry). If a panoramic view is used, a new        “slice” of the panoramic view would be made centered on this new        fixation. If multiple images are used, the image that is        centered most closely to this new fixation would be used.    -   4. With the new view, the system would analyze for the next        salient region.        -   a. The process then repeats (determine fixation, then center            the viewpoint).

EXAMPLES

Several non-limiting examples are provided below which show how theaforementioned systems and methods may be put to practical use.

Example 1 Robustness Calculation Using External Variation

Background: A hotel owner wants to install two digital signs in thehotel lobby. She wants them noticed by customers and can put them in anyof 3 potential locations, resulting in 3 possible configurations of twodigital signs (that is, signs at locations 1-2, 1-3, or 2-3). Robustnessis calculated to recommend the best sign locations to be noticed.

-   1. Capture an image of the hotel lobby using a digital camera,    download the image to a computer capable of running an attention    model. The computer has visual attention modeling software    installed, (for example Koch & Itti), along with Matlab™ (available    from The MathWorks, Natick, Mass.).-   2. Modify the image to include simulated digital signs. Generate    three modified images, each simulating two digital signs such that    all combinations of two signs in the three potential locations are    produced. Use a standard digital photograph manipulation program    such as Photoshop™ (available from Adobe Co., San Jose, Calif.).    Each simulated digital sign is properly scaled and has simulated    content, such as a graphic of the hotel logo. Store the image    regions, as defined by pixel addresses, associated with each of the    three digital sign locations in a file on the computer.-   3. Run the modified images through the attention model. The output    will include the predicted salient regions of the modified images.    Each salient region is compared to the digital sign pixel addresses    stored in Step 2. If the salient region falls within or overlaps    with the stored pixel addresses, then the predicted attention goes    to the desired location. Each of the three modified images shows the    digital signs to be in the top ten fixations, thus confirming that    any of the three locations is a good candidate.-   4. Capture multiple images of the same scene, either using a series    of still photos or using a video and sampling images from the video    stream. Images are taken every 5 minutes over a 16 hour period, thus    capturing the scene external variability resulting from a variety of    lighting conditions and pedestrian movements. The goal is to have    the sign locations robust to these types of variability (lighting    and pedestrian movements). Load these images into the computer and    modify them with simulated digital signs as described in Step 2.-   5. Each modified image from step 4 is analyzed by the attention    model and compared to the stored pixel addresses as described in    Step 3. The series of modified images associated with sign locations    1 and 2 showed that predicted fixations went to both digital sign    locations in 20% of the images. Similarly, locations 1 and 3 had 35%    of fixations going to both sign locations, while locations 2 and 3    had 85% of fixations going to both sign locations. Having signs    installed at locations 2 and 3 resulted in the most robust    configuration, providing the best solution for the hotel. Recommend    this solution to the hotel owner.

Example 2 Robustness Calculation Using Internal Variation

Background: A hotel owner wants to install two digital signs in thehotel lobby. She wants them noticed by customers and can put them in oneof 3 potential locations, resulting in 3 possible configurations of twodigital signs (that is, signs at locations 1-2, 1-3, or 2-3). Robustnessis calculated to recommend the best sign locations to be noticed.

-   1. Capture an image of the hotel lobby using a digital camera,    download the image to general purpose computer capable of running an    attention model. The computer has visual attention modeling software    installed, (for example Koch & Itti), along with Matlab™ (available    from The MathWorks, Natick, Mass.).-   2. Modify the image to include simulated digital signs. Generate    three modified images, each simulating two digital signs such that    all combinations of two signs in the three potential locations are    produced. Use a standard digital photograph manipulation program    such as Photoshop™ (available from Adobe Co., San Jose, Calif.).    Each simulated digital sign is properly scaled and has simulated    content, such as a graphic of the hotel logo. Store the image    regions, as defined by pixel addresses, associated with each of the    three digital sign locations in a file on the computer.-   3. Run the modified images through the attention model. The output    will include the predicted salient regions of the modified images.    Each salient region is compared to the digital sign pixel addresses    stored in Step 2. If the salient region falls within or overlaps    with the stored pixel addresses, then the predicted attention goes    to the desired location. Each of the three modified images shows the    digital signs to be in the top ten fixations, thus confirming that    any of the three locations is a good candidate.-   4. Begin with the basic visual attention model of Koch & Itti, as    specified in Step 1. Specify the number of model variations to    utilize in analyzing the modified images (for example, 100 model    variations). Each visual attention model has three different feature    maps (color, orientation, and luminance); the saliency map is    computed as a weighted combination of each of these maps. The basic    visual attention model sets the weighting parameters for each map as    equal (1, 1, 1). To produce 100 model variations, randomly set the    weighting vector for each model. This is completed by an algorithm    that randomly sets each weight and normalizes the sum of the weights    to 3 (3*(RandWeights/sum(RandWeights)).-   5. Analyze each image by the 100 visual attention model variations    (defined by the 100 random weighting values) and compare the results    to the stored pixel addresses as described in steps 2 & 3. The    series of modified images associated with digital sign locations 1    and 2 shows that predicted fixations go to both digital sign    locations in 20% of the images. Similarly, locations 1 and 3 have    35% of fixations going to both sign locations, while locations 2 and    3 have 85% of fixations going to both sign locations. Having signs    installed at locations 2 and 3 will result in the most robust    configuration for the hotel. Provide this recommendation to the    hotel owner.

Example 3 Scene Optimization

Background: A hotel owner wants to visually optimize her lobby and thecontent displayed on two digital signs. Her specific visual goals arefor customers to notice four target objects: a first and a seconddigital sign, a static graphic sign advertising the hotel restaurant,and the staff behind the check-in counter.

-   1. To generate a score for optimization options, rewards are given    for changes that draw attention to the target objects, and    real-world costs are associated with permissible changes. An    estimated cost in dollars relating to labor and supply costs is    assigned to the potential changes being considered:    -   moving a painting currently located behind the check-in counter:        $100,    -   changing the lighting behind the restaurant sign: $2500, and    -   redesigning content displayed on the two digital signs: $250        dollars each.

Reward values assigned for achieving the visual goals are as follows:

-   -   drawing attention to the two digital signs: $500 each,    -   drawing attention to the restaurant sign: $250,    -   and drawing attention to the staff behind the check-in counter:        $150.

-   2. Capture an image of the existing lobby using a digital camera,    download the image to a computer capable of running an attention    model. The computer has visual attention modelling software, e.g.    Koch & Itti, along with Matlab™ (The MathWorks, Natick, Mass.).

-   3. Modify the image to reflect the changes being considered, so as    to create a plurality of images associated with all possible    combinations of possible changes. Use a standard digital photograph    manipulation program such as Photoshop (Adobe, San Jose, Calif.).    The pixel addresses of the target objects associated with the    customer's visual goals are also specified and stored in memory.

-   4. Analyze each image from Step 3 using the attention model and    compare the salient objects predicted by the model to the stored    pixel addresses for the target objects. A score is computed for each    simulated configuration by subtracting the costs for changes from    the reward values for achieving the visual goals in the modified    image, indicated by an overlap of the predicted visual attention    with the pixel addresses for the target objects. For example, when    attention is allocated to the restaurant sign, using the change of    moving the painting, the score is $250−$100=$150. After analyzing    all of the simulated images, the most cost effective solution found    is to move the painting at a cost of $100 and to modify the color of    one of the pieces of content at a cost of $250 (total cost $350).    These changes allow all of the visual goals to be achieved, yielding    a reward score of $1400 and a total score of $1050.

Example 4 Multi-Perspective Scene Analysis

Background: Continuing from Example 3, recommended changes have beenmade. The hotel owner would like to understand the visual saliency ofeach target object as viewed from multiple perspectives in the hotellobby.

-   1. Four locations of interest, distributed throughout the lobby, are    identified and four digital photos are taken from each location by    turning the camera in 90 degree increments, resulting in a total of    16 images, each image representing one perspective. The images are    photos taken using a digital camera. The images are and downloaded    to a computer capable of running an attention model. The computer    has visual attention modeling software, for example Koch & Itti,    along with Matlab™ (The MathWorks, Natick, Mass.). For each    perspective, the pixel addresses for the target objects are    identified and stored in the computer's memory, and the target    objects tagged with an identifier.-   2. Each of the 16 images from Step 1 are analyzed using the    attention model. For each image, it is determined by the visual    attention model which target objects are visible and which target    objects are predicted to draw attention.-   3. The probability that each target object is visible across all of    the images is computed, and the probability that it is attended to    is also computed. This data is presented in a report to the hotel    owner, providing a better understanding of the visual    characteristics in the lobby from a variety of perspectives.

Note that the example applications for the methods and systems describedherein have broad application beyond the specific applicationsdiscussed. For example, these applications include retail environments.

What we claim is:
 1. A computer-implemented method comprising: definingat least one visual goal for a scene, the scene comprised of a pluralityof objects, the visual goal specifying at least one object in the scenewhich is desired to be predicted to be attended to by a visual attentionmodel; receiving input defining allowable changes to the scene;assigning a monetary cost estimate to at least some of the allowablechanges; automatically modifying, by a processor, some of the pluralityof objects consistent with the defined allowable changes so as toproduce modified scenes; evaluating the modified scenes with a visualattention model; calculating, by the processor, modification costsassociated with the modified scenes based on the monetary cost estimate;and determining, by the processor, based on the evaluation, at leastsome of the modified scenes that achieve the at least one visual goalfor the scene and the modification costs associated with the at leastsome of the modified scenes are lower than a predetermined threshold. 2.The computer-implemented method of claim 1, further comprising:receiving input defining the approximate duration a subject will viewthe scene; and, using this approximate duration to determine which ofthe modified scenes that achieve the at least one visual goal for thescene would continue to achieve the at least one visual goal for thescene considering the approximate duration.
 3. The computer-implementedmethod of claim 1, further comprising: receiving input defining a rewardfor achieving at least one visual goal, and determining which of themodified scenes that achieves the at least one visual goal for the scenealso has the highest reward.
 4. The computer-implemented method of claim3, further comprising: calculating costs associated with at least someof the modified scenes that have been determined not to have achieved atleast one visual goal for the scene, determining at least one modifiedscene that has the highest cost-benefit measure, the cost-benefitmeasure being cost of the change compared with the reward for achievingthe at least one visual goal.
 5. The computer-implemented method ofclaim 1, wherein allowable changes to the scene defines how at leastsome of the plurality of objects may be changed in order to achieve thegoal.
 6. The computer-implemented method of claim 1, further comprising:calculating costs associated with at least some of the modified scenesthat have been determined to achieve at least one visual goal for thescene.
 7. The computer-implemented method of claim 1, furthercomprising: calculating costs associated with at least some of themodified scenes that have been determined not to have achieved at leastone visual goal for the scene.
 8. The computer-implemented method ofclaim 1, further comprising: determining the most cost effectivemodified scene, which is the modified scene that achieves the at leastone visual goal for the scene that has the lowest cost.
 9. Thecomputer-implemented method of claim 8, further comprising: recommendingto a client the allowable changes that give rise to the modified scene.10. The computer-implemented method of claim 9, wherein the allowablechanges concern positioning of objects within the scene.
 11. Thecomputer-implemented method of claim 9, wherein the allowable changesconcern the color of objects within the scene.
 12. Thecomputer-implemented method of claim 1, wherein the visual goaladditionally comprises a sequence in which objects are to be predictedto be attended to.
 13. The computer-implemented method of claim 1,wherein the cost estimate is specified in a currency denomination.
 14. Acomputer-implemented method comprising: defining at least one visualgoal for a scene, the scene comprised of a plurality of objects, thevisual goal specifying at least one object in the scene which is desiredto be not predicted to be attended to by a visual attention model;receiving input defining allowable changes to the scene, which definehow at least some of the plurality of objects may be changed in order toachieve the goal; assigning a monetary cost estimate to at least some ofthe allowable changes; automatically modifying, by a processor, some ofthe plurality of objects consistent with the defined allowable changesso as to produce modified scenes; evaluating the modified scenes with avisual attention model; calculating, by the processor, modificationcosts associated with the modified scenes based on the monetary costestimate; and, determining, by the processor, based on the evaluation,which of the modified scenes achieves the at least one visual goal forthe scene and has the modification cost being lower than a predeterminedthreshold.
 15. A computer system comprising: a processor and memory; ascene optimization module operative to at least: (1) receive inputdefining at least one visual goal for a scene, the scene comprised of aplurality of objects, the objects having properties; (2) receive inputdefining allowable changes, which define how properties can be changed;(3) associate allowable changes with a monetary cost estimate; (4)generate modified scenes by modifying the properties of objectsconsistent with the allowable changes; (5) interact with a visualattention module to determine which modified scenes meet the visualgoal; and, (6) determine the costs associated at least some of themodified scenes based on the monetary cost estimate; a visual attentionmodule operative to evaluate a scene and predict objects within thescene that will tend to attract visual attention.
 16. The computersystem of claim 15, the scene optimization module further operative to:(7) associate rewards with at least one visual goal for the scene, andto determine the reward value associated with at least some of themodified scenes.
 17. The computer system of claim 16, wherein the sceneoptimization module is further operative to determine at least onmodified scene that has an associated cost that is lower than anothermodified scene's associated cost.
 18. The computer system of claim 16,wherein the allowable changes define how at least some of the propertiesof objects may be changed in order to achieve the visual goal.
 19. Thecomputer system of claim 16, further comprising a user interface moduleoperative to provide to a user information indicative of the cost of amodified scene.
 20. The computer system of claim 16, further comprisinga user interface module operative to provide to a user informationindicative of the allowed changes that precipitated a particularmodified scene.
 21. The computer-implemented method of claim 1, whereinthe monetary cost estimate is a relative monetary cost estimate.
 22. Thecomputer-implemented method of claim 14, wherein the monetary costestimate is a relative monetary cost estimate.
 23. The computer systemof claim 15, wherein the monetary cost estimate is a relative monetarycost estimate.